ena_maps: combine PSET non-spatial dims by coordinate label when projecting to a map by tmplummer · Pull Request #3633 · IMAP-Science-Operations-Center/imap_processing

tmplummer · 2026-11-06T16:51:18Z

Change Summary

Overview

Why: project_pset_values_to_map matched only the spatial axis to map pixels; non-spatial dims (e.g. gain_config_id, calibration_prod, esa_energy_step) were combined by array position, with the map variable shaped by the first PSET. A later PSET with different labels, a different order, or a length-1 dim was silently placed by numpy broadcasting — e.g. a PSET with gain_config_id=[1] was added into both gain-config slots of a 2-gain map. Hi L2 gain-config combination (#3619) hits this, and currently works around it by zero-padding every PSET to the full layout, which doubles the projection work and leaves the hazard in place for other callers.

project_pset_values_to_map now combines PSET non-spatial dims into the map by coordinate label instead of array position (PUSH and PULL, rectangular and HEALPix maps). A PSET may hold a subset of the map variable's labels, in any order.
Mismatched dims, labels not in the map, duplicate labels, or positional size mismatches raise ValueError instead of silently broadcasting.
epoch is always combined by position (each PSET has its own epoch label), so Hi and Ultra L2 behavior is unchanged.
Map variables can be pre-initialized in data_1d with the full non-spatial coords (pixel dim last). New variables reuse existing map coords for shared dims.
Label lookups are resolved once per PSET. The subset-label path is faster than zero-padding PSETs to full labels (~18–23 ms vs ~30 ms per PSET in a local benchmark).

Closes #3632. Adopting this in Hi L2 (#3619) is a follow-up.

Behavior changes

A non-spatial dim without coords on the map or PSET is only combined by position when sizes match; a length-1 dim no longer broadcasts across the map (now ValueError). No current caller relies on this, since map variables are created from the first PSET's shape.
Coordinate labels are matched exactly. Float labels that differ slightly (e.g. by 1e-12) were previously added by position; they now raise, with the labels printed at full precision. Ultra is unaffected today because it rebuilds its energy coords from constants.
The PULL path allocates a zero array of shape (*non_spatial, num_points) per value key per PSET (so it can share the slice-based add with PUSH). Negligible at current Hi/Ultra sizes.

File changes

imap_processing/ena_maps/ena_maps.py: new _add_projected_values / _resolve_dim_positions helpers used by both PUSH and PULL; map-variable creation honors existing map dims; PULL masks computed once per PSET; docstring describes label matching and pre-initialization; removed an unreachable value_key check and fixed the value_keys docstring type.
imap_processing/tests/ena_maps/test_ena_maps.py: new TestProjectByCoordinateLabel using a small synthetic rectangular PSET.

Testing

35 new tests: subset/reordered labels vs a zero-padded reference (PUSH/PULL × rect/HEALPix × with/without mask), fast-path equivalence, projection order independence, PSET dim order differing from map, each ValueError case (including duplicate labels on the first PSET), coordinate-less dims staying positional, and a full-precision message for near-equal float labels. 21 of the original 25 fail on the old implementation.
Existing ena_maps, Hi, and Ultra suites pass unchanged (837 passed, 4 skipped).

…a map AbstractSkyMap.project_pset_values_to_map now adds each PSET's binned values into the map variable by non-spatial coordinate label instead of by array position, for both PUSH and PULL. PSETs may hold a subset of the map's labels, in any order. Mismatched dims, unknown labels, and size mismatches raise ValueError instead of silently broadcasting. The epoch dim is combined by position. Map variables can be pre-initialized with the full coords. Closes IMAP-Science-Operations-Center#3632 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Label-not-found error prints labels at full precision and notes that labels are matched exactly. - Positional size-mismatch error states why the dim is positional (no coordinate on map/PSET, or epoch). - Remove unreachable value_key check (KeyError check above covers it). - Fix value_keys type in project_pset_values_to_map docstring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

🟡 Changes recommended

Duplicate energy-step labels bypass validation, and coordinate-less dimensions gain artificial labels that cause incorrect reordering or unexpected errors during later accumulation.

2 open issues

What changed in this PR

This PR adds coordinate-label matching for non-spatial dimensions in ENA map projection, preparing the shared mapping code for Hi L2 gain-configuration combination without zero-padding each pointing set.

Changes:

Aligns subset and reordered labels in PUSH and PULL projection, with positional handling for epoch and unlabeled dimensions.
Supports pre-initialized map layouts and adds validation for incompatible dimensions and labels.
Adds regression tests across rectangular and HEALPix maps, including masks, projection order, and error cases.

File	Description
`imap_processing/tests/ena_maps/test_ena_maps.py`	Adds synthetic pointing-set fixtures and coordinate-label projection tests.
`imap_processing/ena_maps/ena_maps.py`	Adds label-resolution and accumulation helpers, updates initialization, and documents projection behavior.

_{Review effort: Balanced}

…uplicates - When creating a map variable, copy only real PSET coordinates. Dims without a coordinate no longer get artificial 0..N-1 map labels, so they stay combined by position. - Check for duplicate PSET labels before the identical-labels shortcut, so duplicates raise even when the map's labels came from the same PSET. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

lacoak21

This looks really good and thorough to me! Just had a few non blocking questions.

lacoak21 · 2026-11-09T16:53:09Z

+                f"Map variable '{value_key}' has dims {map_var.dims}; the last dim "
+                f"must be the spatial pixel dim '{pixel_dim}'."
+            )
+        if set(pset_dims) != set(map_dims) or len(pset_dims) != len(map_dims):


why is the len() check needed here? On first look I don't think xarray allows for duplicate dim names.

Good question. Claude says that:

xarray (2025.12.0) it only warns: "We do not yet support duplicate dimension names, but we do allow initial construction"

This can and has been updated to just a single test though.

lacoak21 · 2026-11-09T18:16:16Z

+            )
+        if np.array_equal(pset_labels, map_labels):
+            return None
+        positions = map_var.indexes[dim].get_indexer(pset_labels)


Nice! Good to know about get_indexer().

Yeah, I learned about that from this PR too 😃

lacoak21 · 2026-11-09T18:19:45Z

+        map_labels = map_var.coords[dim].values
+        # Check before the equality shortcut, so duplicates are caught even when
+        # the map's labels were copied from this PSET.
+        if len(np.unique(pset_labels)) != pset_size:


Is there a check anywhere that the map doesn't have duplicate labels?

No, there isn't. Great catch! I'll add a check.

tmplummer and others added 2 commits November 6, 2026 09:50

tmplummer requested a balanced review from Copilot November 6, 2026 17:51

Copilot started reviewing on behalf of tmplummer November 6, 2026 17:51

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

Comment thread imap_processing/ena_maps/ena_maps.py Resolved

Comment thread imap_processing/ena_maps/ena_maps.py Resolved

tmplummer requested review from lacoak21, leowerneck and vineetbansal November 6, 2026 18:16

tmplummer self-assigned this Nov 6, 2026

lacoak21 reviewed Nov 9, 2026

View reviewed changes

Address PR feedback

3a44cd6

Navigation Menu

Repository navigation

ena_maps: combine PSET non-spatial dims by coordinate label when projecting to a map - #3633

ena_maps: combine PSET non-spatial dims by coordinate label when projecting to a map#3633
tmplummer wants to merge 4 commits into
IMAP-Science-Operations-Center:devfrom
tmplummer:3632-ena_maps---combine-pset-non-spatial-dims-by-coordinate-label-when-projecting-to-a-map

tmplummer commented Nov 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

lacoak21 left a comment •

edited

Loading

Uh oh!

Uh oh!

lacoak21 Nov 9, 2026

Uh oh!

tmplummer Nov 9, 2026

Uh oh!

lacoak21 Nov 9, 2026

Uh oh!

tmplummer Nov 9, 2026

Uh oh!

lacoak21 Nov 9, 2026

Uh oh!

tmplummer Nov 9, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

tmplummer commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Summary

Overview

Behavior changes

File changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

lacoak21 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lacoak21 Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

tmplummer Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

lacoak21 Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

tmplummer Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

lacoak21 Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

tmplummer Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tmplummer commented Nov 6, 2026 •

edited

Loading

lacoak21 left a comment •

edited

Loading